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FIG. 22 
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RETRIEVAL METHOD AND SYSTEM OF stored on one medium. Even if the medium is found, to 

MULTIMEDIA DATABASE obtain the lecording position of the desired material in it, 

preview work by a regenerator uoit is also necessary. 

BACKGROUND OF THE INVENTION Furthermore, the material is combined with another mate- 

The present invention relates to a daubase system for ^ ^y the editing work and outputted to a new medium, 

preserving and reusing multimedia data such as video, Therefore, video and audio retneved by a user may be 

audio, and animation and more particularly to cm efBcienl "'"'fded in different posiUons of a plurality of media. If, 

retrieval method and retrieval system using the multimedia ^'hen a user, for example, retrieves a video of Mt. Everest 

database * result of retrieval of the video, he has a retrieval 

' , . ... , , . 10 inteotioD that his desired video must be found after the tape 

The following arc typical examples of the pnor art ^^^^^ ^^^.^j^ ^^^^^^ ^ necessary U) 

relating to retrieval of a multimedia database. citeallMt. Everest used in a plurality of media as candidates 

(1) Video on demand (VOD) service as a service for a retrieval object. 

CATV contract household Recently, a trend of storing video and audio in a file of a 

For example, descriptive attribute information of the 15 computer system is developing due to a demand for tapeless 

entire title such as title name, distribution coinpany, and storage. As a result, regeneration from an optional time 

actors and actresses arc given to one video movie and this position can be executed quickly because the feed operation 

attribute information is stored, for example, in a relational fo,. heading like a tape is not necessary. However, as 

database in a tabular form. According to the request (query) mentioned above, when a user has no knowledge on position 

from a viewer, information among this attribute information 20 information of the portion to be found, the preview operation 

which coincides with the conditions are di^layed on the cannot be avoided. 

viewer screen in a tabular fonn or a visual form and the ^ ^^-^ the present invention is to provide a method 

viewer selects the desued Utlc from them A respect that the ^ effectively retrieving an optional recording 

retrieval unit has a large granularity at the unit of UUc is a ^^^-^^^ ^^^^^^^ ^^^^^^^ ^^^^-^ ^^-^^^^^ ^.^j^^^t 

characteristic of this kmd of art. 25 .^q^ring such manual retrieval of a large amount of video 

(2) For example, an OMF (open media framework) inter- and audio materials. 

change specification (Version 2.0) which is one of the ^ ^^^^ ^^^^^ ^^^^^^ comment information^ 

replacement forms of contents dunng creauon m the mul- (hereinafter referred to as an annotation) in which a keyword 

umedia contents creation field is proposed. ^^^^eval is expressed by a character and numeric string 

During creation of one multimedia content comprising for a specific range of video and audio data. Taking notice' 

composed video data and audio data, a model in which a of it, when a user defines an annotation in a specific range 

plurality of tracks are defined for video and audio is used. of the media object, the present invention decides the object 

Each track also comprises a plurality of video sources so that id of the target medium in which the annotation is defined] 

a video A appears until a certain time and a video B follows the specific range, and the annotation information as one) 

it thereafter. Namely, elements such as video and audio object (hereinafter referred to as an annotation object) an'd 

constituting a multimedia title can be operated indcpen- manages a group of these annotation objects. \^ 

^^^^^y- When the user designates an annotation retrieval 

Particularly, in the OMF interchange specification, history condition, the group of annotation objects is retrieved, and 

information can be expressed, for example, as a source from ^ annotation objects having annotation information coincident 

which an element (for example, a part of video data) to be with or similar to the retrieval condition are all extracted, 

incorporated into a certain title is created at the preceding and an annotation list thereof is presented to the user in a 

stage, or such as for example, a video tape having recorded tabular form. When the user designates the desired annota- 

video is captured and fetched at a particular compression tion object in the presented annotation list, the target media 

r^^cd- 45 object is read from the multimedia database and the contents 

However, in the OMFIS, an effective retrieval function for in the specific range are presented to the user. When the 

video and audio and a realization method thereof are not presented contents are not those which are being retrieved 

disclosed. by the user, he stops the regeneration thereof and when he 

designates the next element in the annotation list, the con- 

SUMMARY OF THE INVENTION ^g^ts in the predetermined range are also presented. By 

When a desired portion is to be retrieved from video data doing this, from a large amount of materials of video and 

and audio data stored in a large number of tape media, by audio, the contents in the recording position desired by the 

looking at the label attached to a medium such as a cassette user can be retrieved effectively. 

on which it is recorded depending on an ambiguous memory As an annotation, in addition to the text style such as a 

and loading and previewing a medium which seems to be the ss character and numeric string, when the target medium is 

target one into a regenerator unit, it is ascertained whether it video, a still image (this is referred to as a thumbnail image) 

is the one imder retrieval or not. Or, a full-time staff for of a firame in a specific position (for example, the top) in a 

executing such work is appointed as an organization and the specific range may be \ised. In this case, when a thumbnail 

desire of a retriever is notified to a full-time staff. However, image is defined in addition to the text (keyword) in the 

he also finds a medium according to the same procedure 60 specific range, the annotation list is displayed, for example, 

depending on his memory. If there is no method other than in a tabular form on the screen on which the text as well as 

such a retrieval method, as the number of materials such as the thumbnail image are reduced. By doing this, for an 

video and audio increases, the retrieval becomes difficult and annotation object selected by the user from the list, an actual 

as a result, the valuable material property is kept idle video in the range can be played. 

increasingly. 65 it requires much labor to put annotadons anew to all 

The retrieval intention of a user is often to retrieve a materials and also to another media created by copying the 

specific portion among the materials of video and audio materials. Therefore, according to the present invention, 
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when an annotation is put to a specific range of a certain FIG. 19 is a flow chart of an embodiment of the copied 

media object once, the annotation is automatically applied to section object and annotation object generating process, 

specific ranges of all media objects in the citation relation to piG. 20 is a whole flow chart of an embodiment of the 

it. annotation retroactive application process. 

Concretely, a group of copied section objects defining a ^ pjQ 21 is a drawing for explaining the outline of the 

link relation between media objects of a citation destination annotation retroactive application process, 

and media objects of a citation source are managed. When piQ, 22 is a detailed flow chart of an example of the 

a user executing the editing work performs an operation for citation retrieval process shown in FIG. 20. 
citing a part of a media object as a component of another 

media object, the copied section object is generated and lo DESCRIPTION OF THE PREFERRED 

registered. At the same time, the group of annotation objects EMBODIMENTS 

is searched. When the amiotation for the portion of the media r^^^ embodiments of the present invention will be 

object of the citation source is defined already, the annota- i^i^ed in detail hereunder by referring to the accompa- 

tion IS diverted and an annotation object for the media object ovine drawings 

of the citation destination is automatically generated. When 15 ' . u, i j- c u j- . 

' t a jt f -a c A- FIG. 1 shows a system block diagram of an embodiment 

an annotation is defined for a specific range of a media - . . • . . . i 

object, media objects to which the range is cited are checked °^ ^'^'^w-^^onf^^ '^f n- y 

u - ic-^^ur .'J c J processing unit (CPU) 10 for controlling the whole system 

comprehensively from the aforementioned group of copied ^ , .. . ^Ar:- 

section objects and an annotaUon object for each cited media txcaiMt^ various proasses a memory 20 for stonng 

, . ^ . , ,1 ♦ J ^5 J • .L- • f -50 programs executed by the CPU 10 and data under 

obi ect is automatically generated. By doing this, savmg of 2U ^ » . t • . . i ^ c - 

1 u £ *u • . ?• f . »■ u- . processing, an I/O device 30 such as a keyboard for mpul- 

the labor for the registration operation of annotation obiects \ ^* 7 j- i * • t • 

is realized or j ^j^jg ^ command and data, a display device 40 tor displaying 

results of retrieval and others, a media object management 

BRIEF DESCRIPTION OF THE DRAWINGS table 50, a copied section object management table 60, an 

25 annotation object management table 70, an annotation defi- 

FIG. 1 is a system diagram showing the whole constitu- nition kind management table 80, and a multimedia database 

tion of an embodiment of the present invention. 90 for storing video and audio data and the units are 

FIG. 2 is a drawing showing a constitution example of a connected to each other via a data/control bus 100. The 

media object management table. management tables 50 to 80 comprise, for example, files and 

HG. 3 is a drawing showing a constitution example of a 30 memories. However, the files and memories may be all 

copied section object management table. ^ ^^^^ 

HG. 4 is a drawing showing a constitution example of a Numerals 21 to 24 in the memory 20 indicate processing 

text annotation object management table. programs parUcularly relating to the present mvention and 

...... . . , ^ the processing programs execute the annotation registration 

no. 5 IS a drawmg showing a constitution example of a ^^^^^ amiotation retrieval process for amiotation 

thumbnail image annotation object management table. ^^^^.^^^^ ^^^^^ ^^^^-^^^^ generation process for 

FIG. 6 is a drawing showing a constitution example of a copied section object and annotation object, and the anno- 

user defined annotation object management table. tation retroactive application process. These processes will 

FIG. 7 is a drawing showing a constitution example of an be described in detail later. Numeral 25 in the memory 20 

annotation definition kind management table. 40 indicates a work memory, which is used to set various lists 

FIG. 8 is a drawing showing a constitution example of a tables temporarily which will be described later, 

citation link table. The term of multimedia database 90 is a term for general 

HG. 9 is a whole flow chart of an embodiment of the concept of files and tapes for storing many multimedia items 

annotation registration process. comprising a video object, audio object, animation object, 

CTr^ in- J * -1 J fl u -* £ *i. * *• **• 45 graphic objecl, and text object. For example, when the video 

FIG. 10 is a detailed now chart of the annotation setting . / ^ . • . 1 . . . .1. -j j . . 

„ r.u ' cirf^ o object (video data) is taken into account, the video data is a 

process shown m rIO. 9. ■* c^e ->^r 

row of still images of 15 or 30 frames per second which are 

FIG. 11 is a detailed flow chart of an example of the ^UghUy different from each other. When the still images are 

keyword annotation obtaining process shown in FIG. 10. pi^yed on the display device 40 one after another, they are 

FIG. 12 is a detailed flow chart of an example of the 50 seen as a motion picture of continuous motion for the eyes 

thumbnail image annotation obtaining process shown in of a human as an aSfterimage. There is a time code expression 

FIG. 10. decided by SMPTE for designating the position of each 

FIG. 13 is another detailed flow chart of an example of the fi-ame available, which indicates hour: minute :second: frame 

thumbnail image obtaining process shown in FIG. 12. in the form of HH:MM:SS:FF. With respect to playing from 

FIG, 14 is a whole flow chart of an embodiment of the predetermined time code position, when a correspon- 

annotation retrieval process for annotation retrieval and ^^nce table between the value of time code and the byte 

result of retrieval position on a file is provided, the byte position is found, and 

TTwr^ Iff- j*ijn L*r ir*i. Al^ IS read from the byte position, and when it is passed 

FIG. 15 is a detailed flow chart of an example of the , -j . j • / r j ^^tC 

. ■ t * 1 u • T-Ti- to the video board, it is displayed on the screen. When the 

retneval execution control shown m FIG. 14. ,^ . ' j r *• j- 1 u . 

60 video data is not preserved as a file of a magnetic disk but 

HG. 16 IS another detailed flow chart of an example of the preserved on a beta cam tape as it is, playing from the 

annotation keyword retrieval process shown in FIG. 15. designated time code position can be instructed under the 

FIG. 17 is another detailed flow chart of an example of the machine control for the tape deck. This played video signal 

thumbnail image annotation extracting process shown in (NTS C, etc.) can be displayed on the display terminal via the 

FIG, 15. 55 video board connected to the system. Furthermore, when the 

FIG. 18 is a flow chart of an example of the defined NTSC monitor is directly connected, the output of the tape 

annotation obtaining process used in FIGS. 17 and 19. deck can be also displayed on the monitor screen. Almost in 



04/16/2004, EAST Version: 1.4.1 



6,085, 

5 

the same way, the audio object (audio data) is sampled by 
time and synchronized with the video data in time and voice 
is outputtcd. 

According to the present invention, the storage form (file 
or tape) of video data and audio data is not significant. ^ 
Namely, as management information, information (file name 
or tape ID) for discriminating a medium storing video data 
and a time code common to every medium are used. 
Actually, with respect to the means for playing video data, 
the realization method is different between a file and a tape 10 
as mentioned above. According to the present invention, the 
methods are abstracted as a device capable of feeding to the 
position designated by the time code and performing the 
playing operation from the position. 

FIGS. 2 to 8 show constitution examples of the media 
object management table 50, the copied section object 
management table 60, the annotation object management 
table 70, the annotation definition kind management table 
80, and the citation link table temporarily generated which 
will be described later. ^ 

FIG. 2 shows a constitution example of the media object 
management table 50. The media objea management table 
50 is a table for managing necessary information for each 
media object for many media data stored in the multimedia ^ 
database 90. In FIG. 2, one record 510 corresponds to one 
media object and comprises an object ID 501 of the said 
media data, a title ID 502, a media kind code 503, location 
information 504, and other attribute information of media 
object 505. ^ 

When a media object (media data) is newly stored in the 
media database 90, a user inputs the record 510 fi-om the I/O 
device 30 and registers it in the media object management 
table 50. When a media object is deleted from the media 
database 90, the user inputs the object ID from the I/O 35 
device 30 and designates deletion, and then the record of the 
media object management table 50 is deleted. With respect 
to deletion, it is possible to display the contents of the media 
object management table 50 on the display device 40 in a 
tabular form and designate the record on the display screen. ^ 

FIG. 3 shows a constitution example of the copied section 
object management table 60. The copied section object 
management table 60 is a table for managing the Hnk 
relation of discrimination information of a citation 
destination/citation source medium and the citation position 45 
(in-point and out-point) in the medium when a user execut- 
ing the editing work performs an operation for using (citing) 
a part of a media object as a component of another media 
object. In this case, the citation destination media object is 
called a destination object and the citation source media 50 
object is called a source object. The time code value is used 
for the citation position in the media and the in-point and 
out-point in the destination object are called an in-point time 
code value on destination media and an out -point time code 
value on destination media, and the in-point and out-point in 55 
the source object are called an in-point time code value on 
source media and an out-point time code value on source 
media. 

In FIG. 3, one record 610 corresponds to one copied 
section object and comprises an object ID 601 for indicating 60 
the copied section object uniquely, a destination media 
object ID 602, an in-point time code value on destination 
media 603, an out-point time code value on destination 
media 604, a source media object ID 605, an in-point time 
code value on source media 606, and an out-point time code 65 
value on source media 607. For example, a copied section 
object having an object ID of LOl indicates that the fiames 
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between the time code values of a medium (source object) 
MOl "00:13:28:05" and "00:13:29:00" are used as frames 
between the time code values of another medium 
(destination object) M03 "01:23:30:05" and "01:23:31:00". 
This copied section object is generated whenever a user 
executing the editing work cites a part of a media object as 
a part of another media object and registered in the copied 
section object management table 60. When a media object is 
taken into account in this case, it may become a destination 
object and also a source object at the same time. 

FIGS. 4 to 6 show constitution examples of the annotation 
object management table 70. When a user viewing and 
operating a medium (video, etc.) puts an annotation 
(comment information) to a specific range for convenience 
of later retrieval, the annotation object management table 70 
is a table for managing discrimination information of a 
medium to which the annotation is put, a specific range 
(in-point time code value and out-point time code value), 
and annotation information (character and numeric string, 
thumbnail image, etc.) together as an annotation object. 
Specific ranges to which an annotation is put may be 
partially overlapped with each other. At the registration 
stage, they arc registered in the annotation object manage- 
ment table 70 as different annotation objects. 

In FIGS. 4 to 6, one record 710 corresponds to one 
armotation object and comprises an object ID 701 for 
indicating the annotation object uniquely, a target media 
object ID 702, an in-point time code value 703 and an 
out-point time code value 704 of a specific range to which 
an annotation is put, an annotation kind 705, and annotation 
information 706. In this case, FIG. 4 shows a type (text type) 
that a keyword such as text is used as an annotation, and 
FIG. 5 shows a type (thumbnail style) that still image data 
in which one frame in a specific position in a specific video 
range is reduced or a specific still image (thumbnail image) 
provided separately is used as an annotation, and FIG. 6 
shows a type (user definition type) that a user particularly 
defines an annotation. These may be used properly depend- 
ing on the use. As described later, when an annotation is 
defined in a specific range of an optional media object by 
referring to the copied section object management table 60, 
the annotation object 710 is automatically generated for all 
destination/source objects in the citation relation to the 
range. 

FIG. 7 shows a constitution example of the aimotatioo 
definition kind management table 80. The aforementioned 
annotation object management table 70 is provided at least 
for each annotation kind, and furthermore the same kind 
may be divided into a plurality of tables, and an intrinsic 
name (table name) is given to each management table. The 
aimotation definition kind management table 80 is a table for 
managing the definition kind (type) of each annotation 
object management table and one record 810 corresponds to 
one annotation object management table and comprises an 
aimotation kind 801 of the management table and a table 
name 802. FIG. 7 shows that, for example, an annotation 
object management table having a table name of "AA" is a 
keyword type. 

FIG. 8 shows a citation link table 650 which is tempo- 
rarily generated in the work area 25 of the memory 20 when 
the annotation retroactive application process which will be 
described later is performed. When an annotation is defined 
in a specific range of an optional media object (referred to 
as a noticed medium), the citation link table 650 is a table 
that all destination/source objects in the citation relation to 
the range are listed up from the copied section object 
management table 60 and one record 660 corresponds to one 



04/16/2004, EAST Version: 1.4.1 



6,0! 

7 

listed-up object and comprises a media object ]D 651 and an 
in-point time code value 652 and an out-point time code 
value 653 in a range overlapping with a specific range of the 
noticed medium. 

Next, the annotation registration process 21, the annota- 
tion retrieval process for annotation retrieval and result of 
retrieval 22, the generation process for copied section object 
and annotation object 23, and the annotation retroactive 
application process 24 shown in FIG. 1 will be explained 
sequentially. 

<ANNOTAnON REGISTRATION PROCESS> 

This is a process of registration of an annotation object 
710 newly in the annotation object management table 70 
when a user defines an annotation in a specific range 
(specific time code range) by viewing video and audio data. 
FIG. 9 shows a whole flow chart of the annotation registra- 
tion process 21. 

When a medium is played, a user generally displays the 
contents of the media object management table 50 on the 
display device 40 in a tabular form and instructs a desired 
object ID. By doing this, the said media object is read from 
the multimedia database 90 and displayed on the display 
device 40, for example, in the case of video data. The initial 
state at Step 2100 shown in FIG. 9 indicates it. 

The CPU 10 obtains the record 510 of a media object 
which is an object of the current audience from the media 
object management table 50 (Step 2102) and waits for the 
press of the key by the user (Step 2103). The user visually 
checks the video on the display device 40 and performs 
operations such as play, fast forward, revmid, and pause by 
the I/O device 30 (Steps 2104 to 2107). When he desires to 
put (define) an annotation to a specific video and audio 
range, he presses the aimotalion range designation key at the 
in and out points of the range. The CPU 10 decides the press 
of the annotation range designation key whenever the key is 
pressed (Step 2103) and when the annotation range desig- 
nation key is pressed, the CPU 10 obtains the time code 
value at the position (Step 2108). When the aimotation range 
designation key is pressed two times and the CPU 10 obtains 
the time code values at the in and out points of the range 
(Step 2109), the CPU 10 executes the annotation setting 
process (Step 2110). By this annotation setting process, one 
annotation object 710 is registered in the atmotation object 
management table 70. 

The annotation registration process ends when the user 
presses the end key (Step 2111). However, when the anno- 
tation range designation key is pressed again before it, the 
next annotation registration is executed continuously. 
Hereafter, this operation is repeated as many times as 
required until the end key is pressed. 

FIG. 10 shows a detailed flow chart of the annotation 
setting process 2110 shown in FIG. 9. In FIG. 10, for 
convenience, two types of annotations such as keyword 
(text) and thumbnail image are used. Needless to say, the 
annotation types are not limited to these. For example, to add 
the user definition type, it is desired to add a processing flow 
therefor. 

In the annotation setting process, the screen for inputting 
the annotation information is displayed on the display device 
40 first (Step 2120). When there are a plurality of media 
objects to put an annotation (for example, video and audio), 
the user is instructed the media object to designate and the 
CPU 10 obtains the target media object by a user press of the 
key (Step 2121). Next, the CPU 10 decides whether the 
target media object to annotate is video or not (Step 2122). 
When video is designated, the CPU 10 displays an option of 
whether to add or not the thumbnail image annotation (Step 
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2123) . Thereafter, the CPU 10 waits for the annotation input 
(keyword, thumbnail obtaining designation, etc.) (Step 

2124) . 

To put the keyword annotation to the specific range 
5 designated by the annotation range designation key, the user 
inputs the desired character string (keyword) by the I/O 
device 30, and to add a thumbnail image, the user clicks 
Thumbnail Obtaining which is the option displayed at Step 
2123. 

30 When the user inputs the keyword, the CPU 10 executes 
the keyword annotation obtaining process (Steps 2125 and 
2126). When Thumbnail Obtaining is designated, the CPU 
10 executes the thumbnail image annotation obtaining pro- 
cess (Steps 2127 and 2128), With respect to the annotation 

15 to put to the specific range, there arc three ways available 
such as one of keyword and thumbnail image and both of 
them. A plurality of keywords or a plurality of thumbnail 
images may be defined in one range. The keyword/ 
thumbnail annotation obtaining processes (Steps 2126 and 

20 2128) wUl be described in detail in FIGS. 11 to 13. 

After the annotation obtaining process is performed, the 
CPU 10 decides whether cancel is designated by the user or 
not (Step 2129). When cancel is not designated, the CPU 10 
registers the annotation object produced by the keyword 

25 annotation obtaining process in the text annotation object 
management table 70 as shown in FIG. 4 (Step 2130) and 
registers the annotation object produced by the thumbnail 
image annotation obtaining process in the thumbnail-style 
annotation object management table 70 as shown in FIG. 5 

30 (Step 2131). Thereafter, the CPU 10 erases the screen for 
armotation input (Step 2132) and returns to Step 2111 shown 
in FIG. 9. On the other hand, when cancel is designated by 
the user, the CPU 10 erases the screen for annotation input 
at the point of time and returns to Step 2111 shown in FIG. 

35 9. 

FIG. 11 shows a detailed flow chart of the keyword 
annotation obtaining process shown in FIG. 10 (Step 2126). 
In the keyword annotation obtaining process, the CPU 10 
defines a temporary annotation object in the form shown in 

40 the record 710 in FIG. 4 in the work area 25 of the memory 
20 first (Step 2141), Next, the CPU 10 obtains a unique 
object ID 701 and sets it in the annotation object 710 (Step 
2142), This object ID 701 may be, for example, a serial 
number and is automatically given by the system. 

45 Continuously, the CPU 10 sets the media objects ID 702 to 
put the annotation which are obtained at Steps 2101 and 
2121 and the in-point and out-point time code values 703 
and 704 which are obtained at Step 2108 in the annotation 
object 710 (Steps 2143 and 2144), Furthermore, the CPU 10 

50 sets "keyword** as an annotation kind 705 (Step 2145) and 
sets the character string (keyword) obtained at Step 2124 as 
annotation information 706 (Step 2146). By doing this, one 
annotation object 710 is completed and registered in the 
annotation object management table 70 shown in FIG. 4 at 

55 Step 2130 shown in FIG. 10. 

FIG. 12 shows a detailed flow chart of the thumbnail 
image annotation obtaining process (Step 2128) shown in 
FIG, 10. This is almost similar to that shown in FIG. 11 
basically and a respect that the thumbnail stiU image obtain- 

60 ing process (Step 2156) is added is different. The thumbnail 
image obtaining process wiU be explained in FIG. 13. 

In the thumbnail image annotation obtaining process, in 
the same way as with the keyword annotation obtaining 
process shown in FIG. U, the CPU 10 defines a temporary 

65 annotation object in the form shown in the record 710 in 
FIG. 5 in the work area 25 of the memory 20 first (Step 
2151). Next, the CPU 10 sets a unique object ID 701 such 
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as a serial number in the annotation object 710 (Step 2152) (Step 2203). When the user inputs the retrieval condition, the 

and continuously sets the media objects ID 702 to put the CPU 10 retrieves the armotation object management table 70 

annotation which are obtained at Steps 2101 and 2121, the and extracts an annotation object (710 shown in FIGS. 4 and 

in-point and out-point time code values 703 and 704 which 5) having armotation information (706 shown in FIGS. 4 and 

are obtained at Step 2108, and "thumbnail" as an annotation 5 5) coincident with the retrieval condition (Step 2204). One 

kind 705 one after another (Steps 2153 to 2155). The CPU or a plurality of annotation objects 710 are extracted. The 

10 executes the thumbnail image obtaining process (Step CPU 10 displays through view of the annotation inforaaation 

2156) and sets the name of the file storing the thumbnail 706 of each extracted armotation object 710 on the display 

image obtained by this process as the annotation information device 40 together with the annotation object ID (701 shown 

706 (Step 2157). By doing this, one annotation object 710 is lo in FIGS. 4 and 5) and waits for the next selection by the user 

completed and registered in the thumbnail image annotation (Step 2205). In the text type shown in FIG. 4, the annotation 

management tabic 70 shown in FIG. 5 at Step 2131 shown information 706 is displayed as it is. However, in the 

in FIG. 10. thumbnail image style shown in FIG. 5, a reduced still image 

Next, the thumbnail image obtaining process (Step 2156) of the file shown in the annotation information 706 is 

shown in FIG. 12 will be explained by referring to FIG. 13. 15 displayed. As annotation retrieval, various retrievals such 

A thumbnail image may be directly obtained from an image that only the keyword is simply retrieved and no thumbnails 

in operation or one defined in another file may be used. The are checked, or a list of all thumbnails is extracted, or when 

flow chart shown in FIG. 13 may be applied to any of them. an annotation retrieved by the keyword is given a thumbnail, 

In this case, a file for storing a thumbnail image obtained the thumbnail is also extracted together may be considered, 

from an image in operation is referred to as a thumbnail file. 20 Retrieval execution control at Step 2204 will be described in 

For this thumbnail file or another file, for example, a specific detail in FIGS, 15 to 18. 

area of the multimedia database 90 shown in FIG, 1 may be It is assumed that by viewing the retrieved results on the 

used. The thumbnail file may serve as another file. display device 40, the user selects an annotation object and 

In the thumbnaD image obtaining process, the CPU 10 designates Resultant Audience (play). In this case, the CPU 

selects from what part of the target range (time code range) 25 10 obtains the object ID 702 of the target mediiun from the 

designated by the annotation range designation key to obtain selected annotation object 710 first, checks the object ID 501 

the thumbnail image by a user press of the key first (Step of the media object management table 50 shown in FIG. 2 

2161) and decides whether to obtain it from the head frame on the basis of the object ID, and obtains the location 

of the time code range or from the user designated position information 504 of the said media object (Step 2206). Next, 

other than the head frame (Steps 2162 and 2163). When the 30 the CPU 10 obtains the in and out time code values 703 and 

thumbnail image is obtained from the head frame of the time 704 from the selected annotation object 710 (Step 2207). On 

code range, the CPU 10 makes a reduced still image from the the basis of the location information of the media object and 

video frame of the time code value using the in-point time the in/out time code values which are obtained at Steps 2206 

code value and writes it in the thumbnail image file (Steps and 2207, the CPU 10 reads the media object from the 

2165 and 2167). When the thumbnail image is obtained from 35 multimedia database 90 in the predetermined range and for 

the user designated position other than the head frame, the example, in the case of video, executes the reproduction 

CPU 10 starts the audience process within the time code process such as display on the display device 40 (Step 2208). 

range, obtains the time code of a user designated video When the displayed video is not the one searched by the 

frame newly (Step 2166), makes a reduced still image from user, he stops the current play, selects the next element 

the video frame of the time code value, and writes it in the 40 (annotation object) by viewing the retrieved results, and 

thumbnail image file (2167). In either case, the file name at designates Resultant Audience again. By doing this, the 

this time is set as annotation information of the annotation processes at Steps 2206 to 2208 are repeated. When the user 

object (Step 2169). designates End, the annotation retrieval process for annota- 

On the other hand, when the user does not designate the tion retrieval and result of retrieval ends. Although omitted 

head frame in the time code range and other optional frames, 45 in FIG. 14, as another function, for example, addition of 

the CPU 10 decides that what is defined in another file is function selection for transferring the retrieved result to the 

used (Step 2164), obtains the file name by a user press of the predetermined location may be considered, 

key (Step 2168), and sets the file name as annotation FIG. 15 shows a detailed flow chart of Retrieval execution 

information of the annotation object (Step 2169). When an control (Step 2204) shown in FIG, 14. In this case, when a 

appropriate thumbnail image cannot be found, the file name 50 thumbnail is defined in the coincident portion in keyword 

will not be returned from the user. If this occurs, an retrieval, a process of also extracting the thumbnail is 

exceptional process sudi as cancellation of the thumbnail shown. This is called a keyword/thumbnail combined 

image obtaining process is performed (Step 2170). retrieval process. 

<ANNOTATION RETRIEVAL PROCESS FOR ANNOTA- The CPU 10 obtains input of the retrieval keyword from 

TION RETRIEVAL AND RESULT OF RETRIEVAL> 55 the user by the VO device 30 (Step 2211). The CPU 10 

This is a process of executing the annotation retrieval retrieves the text annotation object management table 70 

under the retrieval condition designated by a user, offering shown in FIG. 4 on the basis of this inputted retrieval 

an annotation list of the retrieval results to the user, and keyword and extracts all annotation objects 710 having a 

playing the specific video scene searched by the user and keyword (character string) coincident with or similar to the 

audio on the basis of it. FIG. 14 shows a whole flow chart so retrieval keyword as annotation information 706 (Step 

of the armotation retrieval process for annotation retrieval 2212). The annotation keyword retrieval process will be 

and result of retrieval. described more in detail in FIG. 16. The CPU 10 rearranges 

The CPU 10 displays a guidance designating a function the text annotation object list exUacted at Step 2212 in the 

selection on the display device 40 (Step 2201) and analyzes ascending order of the object IDs 702 of the object medium 

the function selected by the user (Step 2202). When the user 65 (Step 1213). Next, the CPU 10 rearranges the annotation 

selects Annotation Retrieval, the CPU 10 waits for input of objects (thumbnail armotation objects) 710 of the thumbnail 

the retrieval condition from the I/O device 30 by the user annotation management table 70 shown in FIG. 5 in the 
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ascending order of the object IDs 702 of the object medium is set as an argument of the defined annotation obtaining 

(Step 1214) and extracts the thumbnail annotation (reduced process (Step 2232). The CPU 10 takes out one text anno- 

still image) defined in the time code ranges 703 and 704 tation object from the text table and sets the target media 

overlapping with the text annotation object together with the object ID and the time code range as arguments of the 

annotation information (Step 1215). The extraction of the 5 defined annotation obtaining process (Step 2233) and 

thumbnail annotation is realized by referring to the group of executes the defined annotation obtaining process (Step 

the thumbnail annotation objects rearranged at Step 2214 for 2234). In the defined annotation obtaining process, the CPU 

each extracted text annotation object repeatedly. The thimib- 10 searches the thumbnail annotation list set at Step 2232 on 

nail annotation extraction process will be described more in the basis of the target media object ID and time code range 

detail in FIGS. 17 and 18. Finally, the reference to the text lO which are set at Step 2233 and extracts the thumbnail 

armotation list extracted at Step 2212 and the thumbnail annotation object in which the target media object ID is the 

annotation list extracted at Step 2215 is returned to Step same and a thumbnail is defined in the portion where the 

2205 shown in FIG. 14 and the keyword/thumbnail com- time code range is overlapped. The defined annotation 

bincd retrieval process ends (Step 2216). obtaining process will be described in detail in FIG. 18. The 

At Step 2205 shown in FIG. 14, the CPU 10 displays 15 CPU 10 preserves the thumbnail annotation list extracted by 

through view of the extracted text annotation list of keyword the defined annotation obtaining process in the extracted 

retrieved results, refers to the extracted thumbnail annota- thumbnail annotation list produced in the work memory 27 

tion list at the same time, and when a thumbnail is defined of the memory 20 (Step 2235), then steps the reference point 

in the coincident portion in keyword retrieval, adds and of the text table to the next position (Step 2236), and decides 

displays the thumbnail annotation to the said text annotation. 20 whether there is still another text annotation object to be 

Next, the annotation keyword retrieval process (Step referred to in the text table or not (Step 2237). The CPU 10 

2212) shown in FIG. 15 will be described in detail by takes out the text annotation objects sequentially from the 

referring to FIG. 16. The CPU 10 produces and initializes an text table like this and repeats the defined annotation obtain- 

extracted text annotation list whose style is the same as that ing process. When the objects to be referred to are all gone 

of the text annotation object management table 70 shown in 25 from the text table, the CPU 10 returns the reference to the 

FIG. 4 in the work area 25 of the memory 20 (Step 2221) and extracted thumbnail annotation list to Step 2216 shown in 

locates the pick-up position from the text annotation object FIG. 15 and the thumbnail annotation extraction process 

management table 70 as a head (Step 2222). The CPU 10 ends (Step 2238) 

takes out the annotation object 710 at the head position of FIG. 18 shows a detailed flow chart of the defined 

the text annotation object management table 70 shown in 30 annotation obtaining process. The defined annotation obtain- 

FIG. 4 (Step 2224), collates the keyword given by the user ing process is also used to generate a copied section object 

which is obtained at Step 2211 shown in FIG. 15 with the which will be described later, so that FIG. 18 shows a 

annotation information (keyword) 706 of the annotation generafized representation. 

object 710 (Step 2225), and decides whether they are Initially, the CPU 10 obtains an annotation management 

coincident with or similar to each other (Step 2226). Col- 35 table to be retrieved, a designated medium object ID, and a 

lation may be executed by a well-known text retrieval means designated time code range as arguments (Step 2241). In the 

such as longest coincidence, partial coincidence, or syn- case of the thumbnail annotation extraction process shown 

onym coincidence based on the thesaurus dictionary. When in FIG. 17, the annotation management table to be retrieved 

the keyword (retrieval condition) given fi'om a user is is the thumbnail annotation management table (thumbnail 

coincident with or similar to the annotation information 706 40 annotation list) and the designated medium object ID and 

of the annotation object 710 taken out from the text anno- designated time code range are the text annotation objea ID 

tation object management table 70, the CPU 10 adds the and time code range thereof taken out from he text table 

taken-out annotation object to the extracted text annotation which are obtained at Step 2231. Next, the CPU 10 produces 

hst (Step 2227). Thereafter, the CPU 10 steps to the next and initializes the extracted annotation list (the extracted 

pick-up position in the text annotation object management 45 thumbnail annotation list in the thumbnail annotation extrac- 

table 70 (Step 2228) and repeats the same operation. When tion process) whose style is the same as that of the annota- 

the CPU 10 finishes the retrieval of all the annotation objects tion management table to be retrieved in the work area 25 of 

710 of the text annotation object management table 70 like the memory 20 (Step 2242) and locates the reference point 

this (Step 2223), the CPU 10 returns the reference of the of the annotation management table to be retrieved as a head 

extracted text annotation list to Step 1213 shown in FIG. 15 50 (Step 2243). 

and ends the retrieval of the annotation keyword. Next, the CPU 10 takes out one annotation object from the 

Next, the thimibnail annotation extraction process (Step armotation management table to be retrieved and sequen- 

2215) shown in FIG. 15 will be described in detail by tially decides whether the value of the target media object ID 

referring to FIGS. 17 and 18. is the same as that of the designated medium object ID and 

FIG. 17 shows a whole flow chart of the thumbnail 55 the lime code range thereof is overlapped with the desig- 

annotatioo extraction process. The CPU 10 obtains a refer- nated time code range (Steps 2245 and 2246). When the 

ence to the text annotation list in which the group of text conditions of the two are satisfied, the CPU 10 adds the 

armotation objects extracted by the annotation keyword annotation object to the extracted annotation list (Step 

retrieval process at Step 2213 shown in FIG. 15 is rear- 2247). Thereafter, the CPU 10 steps the reference point to 

ranged in the ascending order of the object IDs of the object 60 the annotation management table to be retrieved next (Step 

media (Step 2231). Hereinafter, in this process, the text 2248) and decides whether there is still another annotation 

armotation list is called a text table. object to be referred to (Step 2249). When there is such an 

Next, the thumbnail annotation list in which the croup of annotation object, the CPU 10 returns to Step 2244, takes out 

thumbnail armotation objects in the thumbnail annotation the next annotation object from the annotation management 

object management table 70 shown in FIG. 5, which is 65 table to be retrieved, and repeats the operations at Steps 

obtained at Step 2214 shown in FIG. 15, is rearranged in the 2245 to 2247. When the armotation objects to be referred to 

ascending order of the object IDs 702 of the object medium are all gone from the annotation management table to be 
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retrieved by cioing this, the CPU 10 returns the reference to source object ID 60S, the source in time code value 606, and 

the extracted annotation list and one defined annotation the source out time code value 607 of the copied section 

obtaining process ends (Step 2250). In the thumbnail anno- management object generated at Step 2305 also as argu- 

tation extraction process shown in FIG. 17, the reference to ments of the defined annotation obtaining process (Step 

the extracted thumbnail annotation list is returned to Step 5 2309). With respect to the defined annotation obtaining 

2235. process, the CPU 10 executes the process shown in FIG. 18 

<GENERA'nON PROCESS FOR COPIED SECTION by putting the annotation object management table at Step 

OBJECT AND ANNOTATION OBJECT> 2308 as an annotation management table to be retrieved, the 

This is a process of generating the copied section object source object ID 605 at Step 2309 as a designated media 

610 shown in FIG. 3 when a user uses a part of a media lo object ID, and the source in/out time code values 606 and 

object as a component of another media object in the editing 607 as designated time code ranges (Step 2310). The CPU 

work and when an annotation object is already defined for 10 decides whether more than one worklog is hit. When 

the portion of the citation source object (source object), there is no worklog to be hit, the CPU 10 returns to Step 

automatically generating an annotation object for the cita- 2306. 

tion destination object (destination object) at the same time 15 On the other hand, when there is more than one worklog 

by diverting the annotation object. to be hit, the CPU 10 takes out the hit annotation objects one 

In the aimotation object generation process, the CPU 10 by one and performs the processes at Steps 2314 to 2317. 

checks the citation range of the media object (soiuxx object) The CPU 10 obtains an overlapping part of the two ranges 

and all annotation objects put to the media object and checks of the source time code ranges 606 and 607 of the hit 

whether an annotation object overlapping with the target 20 annotation object and the copied section management object 

range is defined. When defined, the CPU 10 copies the 610 gene rated at Step 2305 and puts the in/out points thereof 

character string (keyword) as an annotation object of the asTsandTerespectiveIy(Step2314). Next, the CPU 10 puts 

destination media object as it is, copies the storage location the destination media object ID 602 of the copied section 

for a thumbnail image, and sets a range overlapping with the management object 610 generated at Step 2305 as DID (Step 

defined range of the source annotation object within the 25 2315). Next, the CPU 10 selects the text type or the 

applied range of the destination media object. By this thumbnail type as an annotation object management table 70 

process, annotation information set for the original material from the annotation discrimination column of the hit anno- 

can be hit as a candidate in the annotation retrieval shown in tation object 710 and generates and initiahzes a new anno- 

FIG. 14 because the annotation is always cited automatically tation object 710 (Step 2316) Next, for this annotation 

when the material is cited. 30 object, the CPU 10 puts the DID obtained at Step 2315 as a 

FIG. 19 shows a whole flow chart of the generation target media object ID 702, puts Ts and Te obtained at Step 

process for copied section object and annotation object. 2314 as in/out time code values 703 and 704, sets the 

The CPU 10 receives a worklog from the editing system annotation kind 705 and annotation information 706 of the 

(Step 2301). In this worklog, when a user executing the said hit annotation object obtained at Step 2312, and regis- 

editing work performs an operation (copy operation) for 35 ters them as a new annotation object 710 (Step 2317). When 

using a part of a media object as a component of another the process is finished for all the hit annotation objects by 

media object, the discrimination information of the citation doing this (Step 2313), the CPU 10 returns to Step 2306. 

source/citation destination media and the citation range (the Hereunder, the CPU 10 loops Steps 2307 to 2317 until the 

in-point and out-point) are included. The editing system reference record 810 of the annotation definition kind man- 

itself is well-known, so that the explanation thereof will be 40 agement table 80 is all gone. When the reference record 810 

omitted here. is all gone (Step 2306), the CPU 10 returns to Step 2302 and 

Next, the CPU 10 takes out one worklog (Step 2302) and obtains the next worklog. When the worklog represents the 

decides whether it is the end of the worklogs or not (Step copy operation, the CPU 10 executes the processes at Step 

2303). When it is not the end, the CPU 10 decides whether 2305 and the subsequent steps again. When all the worklogs 

the worklog is a worklog representing the copy operation or 45 have been obtained (Step 2303), the generation process for 

not (Step 2304). When the worklog is not a worklog copied section object and annotation object ends, 

representing the copy operation, the CPU 10 steps to obtain- <ANNOTAnON RETROACTIVE APPUCAHON PRO- 

ing of the next woridog. When the worklog is a worklog CESS> 

representing the copy operation, the CPU 10 generates the This is a process of, when an annotation is defined for a 

copied section object 610 shown in FIG. 3 on the basis of the 50 specific range of a media objea, checking a medium com- 

worklog and registers it in the copied section object man- prehensively by which the range is cited and defining an 

agement table 60 (Step 2305). aruiotation also for the cited media object. Namely, when no 

Now, the copied section object generation process for one annotation is put to the original material and the material is 
worklog ends. When the worklog represents the copy cited in various locations, the CPU 10 puts an annotation 
operation, the subsequent annotation object generation pro- 55 later when the annotation is defined for a media object. This 
cess starts continuously. The annotation object generation process may be executed, for example, as a postprocess of 
process is repeated for the defined kinds in the annotation the aimotation registration process shown in FIG. 9. In the 
object management table shown in the aimotation definition same way as with execution of the annotation object gen- 
kind management table 80 shown in FIG. 7. eration process shown in FIG. 19, an annotation can be hit 

In step 2306, the CPU 10 decides whether the annotation 60 as a candidate in the annotation retrieval by this process, 

definition kind management table 80 is finished or not. FIG. 20 shows a whole flow chart of the annotation 

When it is not finished, the CPU 10 obtains one annotation retroactive apphcation process. When an annotation is 

kind and table name from the annotation definition kind defined in a specific range for a media object and the 

management table 80 (Step 2307) and sets the annotation annotation object 710 is set, the CPU 10 puts it as a noticed 

object management table 70 (FIGS. 4, 5, and 6) represented 65 annotation object, obtains the media object ID and time code 

by the table name as an argument of the defined annotation range thereof (Step 2401), furthermore obtains the aimota- 

obtaining process (Step 2308). Next, the CPU 10 sets the tion Idnd and annotation information (Step 2402), and starts 
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the citation retrieval process. In the citation retrieval 
process, the CPU 10 retrieves the copied section object 
management table 60 on the basis of the noticed media 
object ID and time code range and checks a medium 
comprehensively by which the range is cited (Step 2403). 5 
The citation retrieval process will be described in detail by 
referring to FIGS. 21 and 22. By the citation retrieval 
process, the CPU 10 obtains the citation link table 650 as 
shown in FIG. 8 for the group of hit media objects (Step 
2404). The CPU 10 takes out the media objects one by one lo 
from the citation link table 650 (Step 2405), produces one 
annotation object 710 from the annotation kind and anno- 
tation information obtained at Step 2402 and the media 
object ID (651 shown in FIG. 8) and the in/out time code 
values (652 and 653 shown in FIG. 8) of the media object 15 
obtained at Step 2405, and registers it in the annotation 
object management table 70 (FIGS. 4, 5, and 6) according to 
the annotation kind thereof (Step 2407). The CPU 10 repeats 
the process for each media object of the citation link table 
650 and ends the annotation retroactive application process 20 
when there is no unprocessed entry remaining in the citation 
hnk table 650 (Step 2406). 

FIG. 21 is a drawing for explaining the outline of the 
citation retrieval process. To check relations of citations for 
the noticed range (time code range) of a media object M, the 25 
destination media object cited from the range is checked by 
the copied section object management table 60 first. In FIG. 
21, media objects are represented by Ma and Mb, and zero 
or one destination media object or more are generally 
extracted. Also a source media object when the notice range 30 
of the media object M is cited from another media object is 
checked from the copied section object management table 
60. In FIG. 21, media objects are represented by Mc and Md 
and also zero or one destination media object or more are 
generally extracted. As a result of these processes, a group 35 
of destination media objects and source media objects is 
obtained on the basis of the noticed media object. The CPU 
10 obtains an overlapping range of these media objects and 
produces each record of the citation link table 650 shown in 
FIG. 8. 40 

The CPU 10 puts these processes as a procedure, steps the 
noticed point to Ma, Mb, Mc, and Md next, and processes 
them recursively. However, for example, if, when the pro- 
cess from M to Ma is executed by a recursive call, the 
process is retumed to citation to the origin (from Ma to M), 45 
the process is executed endlessly. Therefore, citation return- 
ing to the origin like this is excluded. Exclusion designation 
is carried out by designating the value of a media object ID 
651. When processes executing a recursive call are all 
finished by doing this and the original process is returned, 50 
the citation link table 650 regarding all relations of citations 
is obtained, 

FIG. 22 shows a detailed flow chart of the citation 
retrieval process (Step 2403) shown in FIG. 20. The CPU 10 
produces and initializes the citation link table 650 shown in 55 
FIG. 8 in the work area 25 of the memory 20 (Step 2410). 
The CPU 10 searches the copied section object management 
table 60 shown in FIG. 3 and takes out copied section objects 
610 in which the source media object ID 605 coincides with 
the noticed media object ID (Step 2411). However, when the 60 
object ID 601 of the referred copied section object 610 is an 
excluded object ID whose parameter is designated, the CPU 
10 excludes the copied section object 610. Next, the CPU 10 
takes out copied section objects in which the source time 
code ranges 606 and 607 are overlapped with the noticed 65 
time code range among the copied section objects 610 taken 
out at Step 2411 (Step 2412). With respect to the copied 



section objects 610 restricted at Step 2412, the CPU 10 
produces a record 660 in which the destination object ID 602 
thereof is put as a media object ID 651 and values converted 
to the time code range in the destination object are put as 
new time code ranges 652 and 653 on the basis of the 
overlapping ranges between the noticed time code range and 
the destination time code ranges 606 and 607, and adds it to 
the citation link table 650 (Step 2413). By doing this, for 
example, for the noticed medium M shown in FIG. 21, the 
objects of the destmation media Ma and Mb thereof are 
registered in the citation link table 650. 

Next, the CPU 10 searches the copied section object 
management table 60 again and takes out copied section 
objects 610 in which the destination media object ID 602 
coincides with the noticed media object ID (Step 2414). 
However, also in this case, when the object ID 601 of the 
referred copied section object 610 is an excluded object ID 
whose parameter is designated, the CPU 10 excludes the 
copied section object 610. Next, the CPU 10 takes out 
copied section objects in which the destination time code 
ranges 603 and 604 are overlapped with the noticed time 
code range among the copied section objects 610 taken out 
at Step 2414 (Step 2415). With respect to the copied section 
objects 610 restricted at Step 2415, the CPU 10 produces a 
record 660 in which the source object ID 605 thereof is put 
as a media object ID 651 and values converted to the time 
code range in the source object are put as new time code 
ranges 652 and 653 on the basis of the overlapping ranges 
between the noticed time code range and the destination 
time code ranges 603 and 604 and adds it the citation link 
table 650 (Step 2416). By doing this, for example, for the 
noticed medium M shown in FIG. 21, the objects of the 
source media Mc and Md thereof are registered in the 
citation link table 650. 

The CPU 10 puts the processes at Steps 2411 to 2416 as 
a procedure, preserves the media objects in the citation link 
table 650, steps the noticed media object until all the objects 
are processed, and calls and executes the citation retrieval 
process at Steps 2411 to 2416 recursively (Steps 2417 to 
2412). The parameter setting at Step 2420 is an exclusion 
designation. When processes executing a recursive call are 
all finished by doing this and the original process is returned, 
media objects of all relations of citations for the noticed 
media object are registered in the citation link table 650. 

By the multimedia database retrieval method and system 
of the present invention, the good results indicated below 
can be obtained. 

(1) When an annotation object when a user defines an 
annotation in a specific range of a media object is managed 
and an annotation list coinciding with the retrieval condition 
in annotation retrieval is presented to the user in a tabular 
form, a material having the contents in the desired recording 
position intended by the user can be effectively retrieved 
among a large amoimt of video and audio materials. 

(2) When reduced images are displayed in a tabular form 
as a group of retrieval results of annotation objects using still 
image data of one frame in a specific position of the video 
range and still image data provided beforehand in addition 
to a text (keyword) such as a character and numeric string as 
an annotation, a reduced image of the contents desired by a 
user can be obtained immediately from a video scene 
remembered by him. 

(3) When an annotation is put once and an annotation 
object is automatically generated for a specific range in the 
link relation to all media objects including the media object 
thereof, labor saving for the registration work of annotation 
objects is realized. 
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What is claimed is: 

1. A data management method in a database system 
storing a plurality of objects and first link information 
representing reference relationships among said plurality of 
objects, said method comprising the steps of: $ 

generating and storing second link information indicating 
a link with retrieval information and an object for 
retrieving the object; 

delecting a change of said first link information; 

generating retrieval information indicated by a link in said lo 
second link information given to a source object of the 
changed first link information, if the first link informa- 
tion is changed; and 

generating and storing changed second link information 
indicating a link with said retrieval information and a 
destination object of said changed first link informa- 
tion. 

2. A data management method according to claim 1, 
further comprising the steps of: 

generating a copy of the retrieval information indicated by ^ 
a link in said second link information given to the 
source object of the changed first link information; and 

generating and storing the changed second link informa- 
tion indicating a link with the copied retrieval infor- 
mation and the destination object of said changed first 
link information. ^ 

3. A data management method according to claim 1, 
wherein said object comprises data corresponding to a 

, partial time span when said object has an attribute of time. 

4. A data management method according to claim 1, 
wherein said object is an object having a time axis; and ^0 

said first link information includes a start point and an end 
point on the time axis of the object having said time 
axis. 

5. A data management method in a database system 
storing a plurality of objects and first link information 
representing reference relationships among said plurality of 
objects, said method comprising the steps of: 

generating and storing second link information indicating 
a link with a retrieval information and an object for 
retrieving the object; ^ 

detecting a change of said first link information; 

generating retrieval information indicated by a link in said 
second link information given to a destination object of 
the changed first link information, if the first fink 
information is changed; and 

generating and storing changed second link information 
indicating a link with said retrieval information and a 
source object of said changed first link information. 

6. A data management method according to claim 5, 
further comprising the steps of: 

generating a copy of the retrieval information indicated by 
a link in said second link information given to the 
destination object of the changed first link information; 
aiid 55 

generating and storing the changed second link informa- 
tion indicating a link with the copy of the retrieval 
information and the source object of said changed first 
link information. 

7. A data management method according to claim 5, eo 
wherein said object comprises data corresponding to a 
partial time span when said object has an attribute of time. 

8. A data management method according to claim 5, 
wherein said object is an object having a time axis; and 

said first link information includes a start point and an end 6S 
point on the time axis of the object having said time 
axis. 
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9. A data management method in a database system 
storing a plurality of objects and first link information 
representing reference relationships among said plurality of 
objects, said method comprising the steps of: 

generating and storing second link information indicating 
a link with retrieval information and an object for 
retrieving the object; 

detecting a change of said second link information; 

selecting an object indicated by a link in said first link 
information corresponding to an object indicated by a 
link of the changed second link information, if the 
second link information is changed; and 

generating and storing second link information indicating 
a link with the retrieval information indicated by a link 
of the changed second link information. 

10. A data management method according to claim 9, 
further comprising the steps of: 

generating a copy of the retrieval information indicated by 
a link in said changed second link information; and 

generating the second link information indicating a link of 
an object indicated by a link of said first link informa- 
tion corresponding to an object indicated by a link of 
the changed second link information, and the copied 
retrieval information. 

11. A data management method according to claim 9, 
wherein said object comprises data corresponding to a 
partial time span when said object has an attribute of time. 

12. A data management method according to claim 9, 
wherein said object is an object having a time axis; and 

said first link information includes a start point and an end 
point on the time axis of the object having said time 
axis. 

13. A computer readable storage medium storing a data 
management program, the program being executable in a 
database system storing a plurality of objects and first link 
information representing reference relationships among said 
plurality of objects, said data management program com- 
prising the steps of: 

generating and storing second link information indicating 
a link with retrieval information and an object for 
retrieving the object; 

detecting a change of said first link information; 

generating retrieval information indicated by a link in said 
second link information given to a source object of the 
changed first link information, if the first fink informa- 
tion is diarged; and 

generating and storing changed second link information 
indicating a hnk with said retrieval information and a 
destination object of said changed first link informa- 
tion. 

14. A computer readable storage medium storing a data 
management program, the program being executable in a 
database system storing a plurality of objects and first link 
information representing reference relationships among said 
plurality of objects, said data management program com- 
prising the steps of: 

generating and storing second link information indicating 
a link with retrieval information and an object for 
retrieving the object; 

detecting a change of said first link information; 

generating retrieval information indicated by a link in said 
second link information given to a destination object of 
the changed first link information, if the first link 
information is changed; and 

generating and storing changed second link information 
indicating a link with said retrieval information and a 
source object of said changed first link information. 
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15. A computer readable storage medium storing a data 
management program, the program being executable in a 
database system storing a plurality of objects and first link 
information representing reference relationships among said 
plurahty of objects, said data management program com- 5 
prising the steps of: 

generating and storing second link information indicating 

a link with retrieval information and an object for 

retrieving the object; 
detecting a change of said second link information; 
selecting an object indicated by a hnk in said first link 

information corresponding to an object indicated by a 

hnk of the changed second link information, if the 

second link information is changed; and 
generating and storing second hnk information indicating 

a hnk with the retrieval information indicated by a link 

of the changed second link information. 

16. A data management program in a database system 
storing a plurality of objects and first link information 20 
representing reference relationships among said plurahty of 
objects, said data maoagemenl program comprising the 
codes for: 

generating and storing second fink information indicating 
a hnk with retrieval information and an object for 25 
retrieving the object; 

detecting a change of said first link information; 

generating retrieval information indicated by a hnk in said 
second link information given to a source object of the 
changed first hnk information, if the first link informa- 
tion is changed; and 

generating and storing changed second Unk information 
indicating a Imk with said retrieval information and a 
destination object of said changed first link informa- 
tion. 



17. A data management program in a database system 
storing a plurahty of objects and first link information 
representing reference relationships among said plurahty of 
objects, said data management program comprising the 
codes for: 

generating and storing second hnk information indicating 
a hnk with retrieval information and an object for 
retrieving the object; 

detecting a change of said first link information; 

generating retrieval information indicated by a link in said 
second link information given to a destination object of 
the changed first Imk information, if the first hnk 
information is changed; and 

generating and storing changed second link information 
indicating a hnk with said retrieval information and a 
source object of said changed first fink information. 

18. A data management program in a database system 
storing a plurahty of objects and first link information 
representing reference relationships among said plurahty of 
objects, said data management program comprising the 
codes for: 

generating and storing second hnk information indicating 
a hnk with retrieval information and an object for 
retrieving the object; 

detecting a change of said second link information; 

selecting an object indicated by a hnk in said first hnk 
information corresponding to an object indicated by a 
hnk of the changed second link information, if the 
second link information is changed; and 

generating and storing second hnk information indicating 
a link with the retrieval information indicated by a hnk 
of the changed second hnk information. 



04/16/2004, EAST Version: 1.4.1 



